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Abstract 

Findings from three field tests evaluations of early childhood intervention practitioner performance checklists and three 
parent practice guides are reported. Forty-two practitioners from three early childhood intervention programs reviewed 
the checklists and practice guides and made (1) social validity judgments of both products, (2) judgments of the 
compatibility of the checklists and practice guides, and (3) suggestions for improving the intervention products and 
materials. Results showed that practitioner feedback and suggestions yielded valuable information for improving the 
products where changes made in response to the practitioners’ social validity ratings and suggestions from the first field 
test had discernible effects on judgments and feedback of revised products. The importance of striving to develop 
intervention products and materials that are judged as socially important and acceptable is described. 

Keywords: social validity, performance checklists, practice guides, practitioner appraisals, product improvement 

1. Introduction 

The extent to which intervention practices in general, and early childhood intervention practices in particular, are judged 
positively by practitioners and the parents with whom they work is dependent, in part, on the perceived importance and 
value of the practices (Kazdin, 1999). The social importance and acceptability of intervention practices in turn would be 
expected to be related to practitioner use of the practices with fidelity (Foster & Mash, 1999). The importance and 
acceptability of intervention practices are described as social validity (Turan & Meadan, 2011). Social importance is 
most often assessed in terms of consumer or end-user judgments of the utility of both intervention practices and the 
intended outcomes of the practices (e.g.. Using this practice would be worth my time and effort), whereas social 
acceptance is most often assessed in terms of judgments of how well an intervention practice can improve everyday life 
(e.g.. This practice can easily be used to improve my child’s behavior). As noted by Strain et al. (2012), no practice, no 
matter its evidence base, is likely to be used by practitioners or parents if the practice itself is not viewed as socially 
valid and worth the time and effort to adopt and use. 

1.1 Purpose 

The purpose of the field tests evaluations described in this paper was to obtain early childhood intervention practitioner 
social validity judgments of performance checklists and parent practice guides developed at the Early Childhood 
Technical Assistance (ECTA) Center (ectacenter.org/decrp). Recently revised early intervention and early childhood 
special education recommended practices were used as the foundations for developing the checklists and practice guides 
(Division for Early Childhood, 2014). Early childhood intervention includes the provision of different types of supports 
and resources to young children birth to 6-8 years of age and their families to improve child, parent, and family 
functioning (Shonkoff & Meisels, 2000). The term early childhood intervention encompasses early intervention for 
infants and toddlers with identified disabilities or developmental delays (Dunst & Espe-Sherwindt, in press), early 
childhood special education for preschoolers with identified disabilities (Reichow, Boyd, Barton, & Odom, 2016), early 
years education for infants, toddlers, and preschoolers who are at-risk for poor developmental outcomes due to family 
circumstances (Burger, 2010), and early childhood education for all young children birth through eight years of age 
(Copple & Bredekamp, 2009). 

1.2 Recommended Early Childhood Practices 

There are eight Division for Early Childhood (DEC) recommended practices topic areas (assessment, environment, 
family, instruction, interaction, leadership, teaming and collaboration, transitions) where each topic area includes 
between 2 and 13 practices that differ considerably in their formatting, specificity, and internal coherence. Content 


176 



Journal of Education and Training Studies 


Vol. 5, No. 3; March 2017 


analyses of the recommended practices for each topic area were used to identify internally consistent sets of practice 
characteristics for each topic area where the characteristics were used to develop operationalized sets of intervention 
practice indicators. For example, the five DEC interaction practices were used to develop four checklists (adult-child 
interactions, child social-communication interactions, child social-emotional competence, child-child interactions) 
where each checklist (e.g., adult-child interactions) was designed to promote and strengthen child interactional behavior 
(e.g., using social games to promote turn taking skills). 

1.3 Performance Checklists 

The performance checklists were developed using a conceptualization-operationalization-measurement framework 
(Dunst, Trivette, & Raab, 2015) for operationalizing the DEC recommended practices (Division for Early Childhood, 
2014). The checklists are all formatted in the same way to facilitate practitioner understanding and use of the 
operationalized practice indicators. Research indicates that material organized and formatted in similar ways facilitates 
learning, memory, and recall (Crowder, 2015; B. L. Schwartz, 2014). Revisiting and repeated use of the material (i.e., 
checklists and practice guides) improves the efficiency of learning, remembering, and internalization content of the 
material (Bransford et ah, 2003; Druckman & Bjork, 1994). 

Appendix A shows the performance checklist for promoting family capacity to provide a child everyday learning 
opportunities for promoting his or her development. Each checklist includes: (1) a brief description of the purpose of a 
checklist and how it can be used to plan or evaluate interventions, (2) an internally consistent set of operationalized 
practice characteristics, (3) a rating scale for assessing how much or how well the practice characteristics were able to 
use by a practitioner, and (4) space for recording notes. The instructions on each checklist state the purpose of the 
intervention and expected outcome or benefit, how a practitioner can use the checklist to affect changes in parent or 
child behavior, and how practitioners can use the checklist indicators to monitor their use of the checklist indicators. 
The checklist indicators include different elements or key characteristics of a practice that, taken together, operationally 
define an evidence-based or evidence-informed intervention (Dunst, 2016). The rating scale for assessing the use 
(adherence) of the checklist indicators ranges from seldom or never (was able to use the checklist indicator) to (was able 
to use the checklist indicators) most of the time. 

The checklists are used on either an a priori basis to review key practice characteristics and serve as a mnemonic devise 
for remembering important procedural steps or on a post hoc basis to determine how well one was able to use the 
checklist practice characteristics. Gawande (2009) described these as Read-Do and Do-Confirm uses respectively. The 
checklists that were the focus of the field-test evaluations described in this paper all include behavior indicators that 
“serve as concrete reminders of the tasks that need to be performed” to implement a practice with fidelity (Wilson, 
2013). 

Both practitioner and parent practice guides have been developed for each checklist. Each practice guide is formatted in 
the same way. Appendix B shows a practice guide that was developed using the Family Capacity-Building Practices 
Checklist indicators. Each practice guide includes: (1) a description of the purpose and importance of a particular type 
of practice, (2) 5 or 6 ideas, examples, and suggestions for how to implement the practice, (3) a vignette of a 
practitioner or parent using the practice, (4) short video clips of parents or practitioners using the practice, (5) outcome 
indicators for determining if the practice had expected child benefits, and (6) an external link to additional resources for 
similar types of practices. There are both web-based and mobile versions of each practice guide. 

1.4 Social Validity Research 

Practitioner and parent social validity judgments of early intervention practice materials have been found useful for 
informing changes and improvements in different types of intervention materials (e.g., Dunst, Masiello, Meter, Swanson, 
& Gorman, 2010; Dunst, Pace, & Hamby, 2007; Dunst, Trivette, Gorman, & Hamby, 2010; I. S. Schwartz, 1996). Dunst 
et al. (2013), for example, used parents’judgments of the social validity of four socially interactive robots to select the 
one robot that parents found most acceptable and judged most likely to be engaging to their children. The robot in turn 
was found effective for promoting young children’s early communication and language development (Dunst, Hamby, 
Trivette, Prior, & Derryberry, 2013a, 2013b). 

Social validity judgments have also been traced to a number of parent, practitioner, and child outcomes. In a study of 
Head Start teachers, practitioner social validity ratings of practices constituting the focus of professional development 
were found to be related to how engaged the practitioners were in the professional development (Trivette, Raab, & 
Dunst, 2014). In another study of parents’ judgments of the social validity of interest-based child language learning 
practices, results showed that social validity was not only directly related to the parents’ fidelity of use of the practices 
with their children but indirectly related to the rates of changes in child language development mediated by the 
frequency of child engagement in interest-based language learning activities (Dunst, Raab, & Hamby, 2016). Foster and 
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Mash (1999), Strain, Barton, and Dunlap (2012), and Wainer and Ingersoll (2013) as well describe how social validity 
judgments are related to the fidelity of use of intervention practices and outcomes of interest. 

Results from the field tests evaluations of three different practitioner performance checklists and three different parent 
practice guides are reported in this paper. Findings from the first field test were used to make changes and 
improvements in the checklists and practice guides subsequently evaluated in the second and third field tests. The 
changes made in response to practitioners’ social validity ratings and suggestions and recommendations were expected 
to be related to between field test differences for the first vs. second and third field test. We also expected to find: (1) 
changes in the practitioners’ social validity judgments for between field test comparisons and (2) fewer repeated 
suggestions for improving the checklists and practice guides. 

2. Method 

2.1 Participants 

The field test participants were 42 practitioners from early childhood intervention programs in two states and an Early 
Head Start Program in a third state in America. The investigator has collaborated with the directors from all three 
programs on different initiatives, including a number of evidence-based studies. The program practitioners are 
knowledgeable about a wide variety of state-of-the-art and contemporary early childhood intervention practices and 
were considered excellent candidates for objectively evaluating the ECTA Center performance checklists and practice 
guides. 

Table 1 shows the background characteristics of the participants. The majority of participants had bachelors or masters 
degrees. Most participants had degrees in early childhood education, early childhood special education, and special 
education, whereas the other practitioners reported their professional disciplines as speech and language pathology, 
child and family specialists, or early interventionists. The practitioners’ years of experience varied considerably ranging 
from less than one year to more than 20 years. None of these background characteristics are related to the practitioners’ 
social validity judgments (Dunst & Hamby, 2017). 


Table 1. Background Characteristics of the Field-Test Participants 


Respondent Characteristics 

Number 

Percent 

Education Degree 

Associates Degree 

8 

19.0 

Bachelors Degree 

14 

33.3 

Masters Degree 

16 

38.1 

Doctorate Degree 

2 

4.8 

Not Reported 

2 

4.8 

Professional Discipline 

Early Childhood 

24 

57.1 

Early Childhood Special Education/Special Education 

10 

23.8 

Other a 

6 

14.3 

Not Reported 

2 

4.8 

Years of Experience 

< 1 

1 

2.4 

2-5 

10 

23.8 

6-10 

11 

26.2 

11-15 

6 

14.3 

16-20 

7 

16.7 

21 + 

5 

11.9 

Not Reported 

2 

4.8 


a Speech and language pathology, child and family specialists, and early interventionists. 


2.2 Field-test Evaluation Survey 

Foster and Mash’s (1999) framework for developing social validity indicators for assessing the importance and 
acceptability of intervention practices, and the intended outcomes of the practices, was used to develop the field test 
survey items. The social validity items for both the intervention practices and expected outcomes of the practices were 
adopted from those used in other studies (e.g., Dunst et ah, 2007; Dunst, Trivette, et ah, 2010; Trivette, Dunst, Masiello, 
Gorman, & Hamby, 2009). 

The field test surveys each included three sections with each section including four social validity items. The first 
section included social validity questions about the performance checklist (e.g., the checklist items are easy to 
understand and follow); the second section included social validity questions about the practice guides (e.g., the practice 
guide would be worth the time and effort to use); and the third section included social validity questions about how well 
the checklist characteristics were incorporated into the practice guides (e.g., the majority of the checklist characteristics 
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are included in the practice guide activities). Participants rated each social validity item on a 5- point scale ranging from 
Do Not Agree At All to Agree A Great Deal with each social validity statement. Each participant’s social validity 
responses were totaled to obtain a summated score for each survey section (Spector, 1992). 

The respondents were also asked open-ended questions about both the checklists and practice guides. The open-ended 
checklist questions asked for suggestions to improve the: (1) checklist instructions, (2) checklist indicators, (3) 
self-evaluation scale, and (4) any other suggestions to improve the checklists. The open-ended practice guide questions 
asked for suggestions to improve the: (1) practice guide format, (2) practice guide activities, (3) child outcomes, (4) 
video examples, and (5) any other suggestions to improve the practice guides. The participants’ responses to each 
open-ended question were coded as no response or no suggestions (0), responses without a specific suggestion to 
improve the checklist or practice guide (e.g., the practice guide video included good examples of social games) (1), or 
responses with a specific suggestion to improve a checklist or practice guide (e.g., video captions of the activities would 
make the examples easier to follow) (2). The field test surveys were completed online using Qualtrics Survey Software. 

2.3 Procedure 

Each field test entailed an email invitation sent to the Program Director of each participating program which included 
information about a performance checklist, practice guide, the field test survey, and instructions for how to review the 
products and complete the survey. A URL link to the survey was embedded in the email. The Program Directors were 
asked to forward the invitation to their staff to decide if they wished to participate in the field test. Participation in the 
field test was voluntary and no personal identification information was requested in order to maintain anonymity and 
confidentiality. Results from the field tests were provided to the Program Directors in only an aggregate and not a 
program specific manner to conceal any information about a particular program. The field-test evaluations were 
considered exempt from human subjects review because they involved practitioner willingness to provide feedback on 
materials designed for routine, every-day early childhood intervention (U. S. Department of Health and Human Services, 
2009). 

Table 2 shows the checklists and parent practice guides that were the focus of practitioner judgments and suggestions. 
The Adult-Child Interaction Checklist included practice indicators for engaging a child in interactive episodes to 
promote child interactional competencies. The Social Games parent practice guide for the checklist included lap games 
(e.g., peek-a-boo, so-big) that parents could use to engage their children in your turn-my turn interactions. The Natural 
Environment checklist included practice indicators for using everyday activities and routines as sources of child 
learning opportunities. The It’s Natural parent practice guide for the checklist included activities for how to increase 
child engagement in everyday learning activities. The Naturalistic Instruction Checklist included practice indicators for 
responding contingently to child behavior and encouraging elaborations in child behavior competencies. The Learning 
Comes Naturally practice guide for the checklist included activities a parent could use to (1) engage a child in everyday 
activities and (2) respond to child engagement in ways supporting and strengthening child competence. 

Practitioners’ judgments, comments, and suggestions from the first field test were used to make changes and 
improvements on both the checklists and practice guides which were the focus of practitioner feedback on the second 
and third field tests. The changes to the checklists included clarifying the fact that the checklists were intended to be 
used by practitioners and not by parents, improving the checklist instructions for doing a self- evaluation, the wording 
of the checklist practice characteristics (indicators) to improve understandability, and changing the terminology for the 
checklist characteristics to improve meaning and intent. The changes to the practice guides included adding captions to 
the video examples of the practices, adding additional activities to the practice guides, providing suggestions about 
adaptations to the activities and practices, and improving the specificity of the child outcomes. 

Table 2. Performance Checklists and Practice Guides That Were the Focus of Practitioner Feedback and Suggestions 


Field 

Test 

DEC 

Topic Area 

ECTA Center Products 


Performance Checklist 

Practice Guide 

1 

Interaction 

Adult-Child Interactions 

Social Games 

2 

Environment 

Natural Environments 

It’s Natural 

3 

Instruction 

Naturalistic Instruction 

Learning Comes Naturally 


2.4 Methods of Analysis 

Three 3 Between Field Test ANOVAs with the summated social validity scores as the dependent measures were used to 
evaluate differences in the practitioners’ ratings. Each ANOVA included a priori orthogonal contrasts comparing the 
second and third field test ratings to the ratings from the first field test in order to obtain the correct denominator term 
for computing the mean difference effect size for the between field test contrasts. Cohen’s d effect sizes for the between 
field test contrasts were used as the primary metric to determine if the changes had discernible effects on the 
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practitioners’ social validity judgments. Effect sizes rather than statistical significance were used for substantive 
interpretation because effect sizes and not /^-values are the best estimates of the magnitude of the difference between 
two groups or comparisons (Coe, 2002). Effect sizes between .20 and .49 are considered small, effect sizes between .50 
and .79 are considered medium, effect sizes between .80 and 1.19 are considered large, and effect sizes of 1.20 or higher 
are considered very large (Cohen, 1988). 

The percent of indicators rated a 4 or 5 on the 5-point scale on the field test surveys were computed for ascertaining if 
the social validity ratings reached a generally agreed upon level (85%) of acceptability and importance (e.g.. Carter, 
2009; Finn & Sladeczek, 2001; Meadan, Ostrosky, Zaghlawan, & Yu, 2009; Strohmeier, Mule, & Luiselli, 2014). The 
percent of indicators rated as acceptable and important were computed separately for the performance checklists, 
practice guides, and the compatibility of the checklists and practice guides. A consumer sciences perspective of social 
validity appraisals indicates that when at least 85% of items on a 5-point scale are rated a 4 or 5, those judgments are 
associated with continued use of a service, product, or practice (Bruder & Dunst, 2015; Dunst & Trivette, 2005; 
Reichheld, 2003). 

A series of between field test chi-square analyses was used to evaluate the participants’ suggestions to improve the 
checklists and practice guides. The dependent measure was the percent of participants who made suggestions for the 
four checklist questions and five practice guide questions. Fewer suggestions were expected on the second and third 
field tests compared to the first field test. Each chi-square analysis included a 2 Between Field Test (1 vs. 2 + 3) X 2 
Response (Suggestion vs. No Suggestion) contrast where the chi-square test results were used to compute a Cohen’s d 
effect size for between field test differences (Dunst & Hamby, 2012). 

In addition to the effect size analyses, we computed the improvement indices for the changes on the checklists and 
practice guides for evaluating the practical importance of the changes (What Works Clearinghouse, 2014). This is a 
measure of the improvements in the checklists and practice guides as a result of the changes made in response to the 
participants’ suggestions. An improvement index can vary from -50 to + 50, where positive indices favoring the second 
and third field tests indicate that the changes made based on the results from the first field test improved the checklists 
and practice guides. ZCalc was used to compute the improvement indices (Neill, 2006). 

Both sets of quantitative results were supplemented by content analyses of the responses to the open-ended questions. 
This was done to determine; (1) if the suggestions made during the first field test were mentioned in the second and 
third field tests, (2) identify new suggestions not mentioned in the first field test, and (3) identify additional changes to 
further improve the ECTA Center performance checklists and practice guides. 

3. Results 

3.1 Social Validity Judgments 

Table 3 shows the mean social validity ratings from the three field tests and Table 4 shows the results from the three 
between field test ANOVAs and the mean difference effect sizes, and improvement indices for the first field test vs. 
second and third field tests. The Cohen’s d effect sizes were small to medium where all the metrics indicated that there 
were increases in the practitioners’ social validity ratings as a function of changes made in response to the first field test 
results. Inspection of the mean scores for the three sets of ratings show, with the exception of the natural environment 
checklist mean scores, that the mean scores for the other two natural environment scores and all of the naturalistic 
instruction mean scores were larger than those for the first field test social validity scores. The differences were most 
pronounced in terms of changes in the social validity judgments for how well the checklist indicators were incorporated 
into the practice guides (d = .44) and the importance and acceptability of the practice guide activities (d = .23). 

Table 3. Mean Social Validity Scores and Standard Deviations (SD) for the Interaction, Environment, and Instruction 
Performance Checklists and Practice Guides 


Between Product Descriptive Statistics 


Practitioner Ratings 

Interaction 

Environment 

Instruction 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Performance Checklists (PC) 

16.62 

2.22 

16.09 

3.62 

17.83 

1.95 

Practice Guides (PG) 

17.51 

2.89 

17.98 

2.03 

17.94 

2.29 

PC/PG Relationship 

16.39 

2.18 

17.49 

2.23 

17.02 

2.71 
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Table 4. ANOVA Results, Effect Sizes and Improvement Indices for the Interaction (1) vs. Environment (2) and 
Instruction (3) Performance Checklists and Practice Guides 



Between Field Test Comparisons 

Field Test 1 

vs. 2 + 3 

Practitioner Ratings 

E-test 

p-value 

Effect Size 

Improvement 

Index 

Performance Checklists (PC) 

1.80 

.178 

.16 

6 

Practice Guides (PG) 

0.15 

.860 

.23 

9 

PC/PG Relationship 

.571 

.572 

.44 

17 


The improvement indices associated with the effect size differences ranged between 6% and 17% for the changes in the 
mean social validity scores. These results indicate that there were small but nonetheless practically significant 
improvements from the first to second and third field tests where the improvement indices were larger for the changes 
made to the practice guides and for how well the checklists and practice guides were conceptually and procedurally 
related. These findings indicate that the changes made in response to the first field test results were primarily limited to 
the participants’ judgment of the practice guides and how well the checklist indicators were incorporated into the 
practice guides. 

Figure 1 shows the percent of social validity indicators that were rated a 4 or 5 for the first field test vs. second and third 
field tests. An acceptable level of social validity was reached on all six measures. Thus, despite small between field test 
differences (Table 3), the majority of practitioners making judgments of the checklists and practice guides rated the field 
test products as both socially acceptable and important as evidenced by the percent of items rated a 4 or 5 on the 5-point 
social validity scale. 

3.2 Practitioner Suggestions 

The percentage of practitioner suggestions for improving different features of the checklists and practice guides for the 
first field test vs. the second and third field tests are shown in Table 5. Eight of the nine chi-square results were 
statistically significant, where there were fewer suggestions for improvements after changes were made based on the 
first field test results. The Cohen’s d effect sizes for the between field test contrasts for all but two comparisons were 
large or very large. These results indicate that there were discernible improvements in the checklists and practice guides 
as evidenced by a fewer number of suggestions by the second and third field test participants. 


100 


95 

a: 90 

o 

£ 85 

u 

Q 80 

z 

- 75 

o 

I- 70 

Z 

u 65 
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£ 60 
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Figure 1. Percent of survey items rated as socially acceptable and important by the field test participants 

The practical importance of the changes to the checklists and practice guides is shown in Table 5 in terms of the percent 
of fewer practitioner suggestions after changes made based on the first field test results. The improvement indices for 
the checklists ranged between 29% and 41%, and the improvement indices for the practice guides ranged between 25% 
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and 41% (except for the differences in the suggestions for improving the child outcome indicators). These results 
indicate that the practitioners participating in the second and third field tests made fewer suggested changes compared 
to the suggestions of the first field test practitioners. 

Table 5. Percent of Field Test Respondents Who Made Suggestions for Improving the Performance Checklists and 


Practice Guides 

Survey Items 

Percent of Respondents 

Interaction Env. + Inst. a 

2 

i 

p-value 

Effect 

Size 

Improvement 

Index 

Performance Checklists 







Checklist Instructions 

67 

14 

8.23 

.004 

1.23 

39 

Checklist Indicators 

67 

24 

3.86 

.050 

0.82 

29 

Checklist Rating Scale 

44 

9 

5.43 

.020 

0.90 

32 

Other Suggestions 

38 

0 

9.93 

.002 

1.34 

41 

Practice Guides 







Practice Guide Format 

57 

9 

7.47 

.006 

1.18 

38 

Practice Guide Activities 

58 

8 

11.65 

.001 

1.33 

41 

Practice Guide Videos 

70 

16 

9.67 

.002 

1.24 

39 

Practice Guide Outcomes 

20 

12 

0.37 

.541 

0.21 

8 

Other Suggestions 

25 

4 

3.90 

.068 

0.48 

25 


a Env. = Environment and Inst. = Instruction. 
3.3 Open-Ended Responses 


All of the open-ended questions asked respondents for suggestions to improve some specific aspect of the checklists and 
practice guides. There were, nonetheless, many different positive appraisals of both products without any suggestions 
for improvements. Of the total number of participant responses to the open-ended questions, 28% of the comments on 
the checklists included positive comments, and 17% of the comments on the practice guides included positive 
comments. The positive comments on the checklists included things such as “The checklist offers a wide variety of 
ways to observe and encourage adult-child interactions,” “The checklist items were concise, understandable, and to the 
point,” and “The rating scale allows for a self-evaluation of the practices.” The positive comments on the practice 
guides included things such as “1 thought the format was well designed and easy to follow,” “The activities can be used 
anywhere and anytime with different families,” and “The videos are great...and the captions make it easy to see what 
the practices look like.” 

There were, however, a number of suggestions on the second and third field tests that continued to be mentioned even 
after the changes that were made based on the first field test results. These included: (1) the fact that the checklists are 
intended to be used by practitioners and not by parents and (2) questions about the instructions for how to use the 
checklists. There were also repeated suggestions about the wording of the checklist practice characteristics (e.g., too 
wordy; simplify the language). One of the common suggestions on the second and third field tests was the terminology 
on the checklists. The most frequently mentioned concern was that the checklists included terminology (e.g., contingent 
responsiveness, natural consequences) that some practitioners might not understand. 

A number of practice guide suggestions on the first field test were also made in the second and third field tests. These 
included the wording on the practice guides and not knowing that the “You’ll Know That it’s Working” section of the 
practice guides were the expected child outcomes of the practices. There were also a number of comments about the 
practice guide videos where a few respondents said the videos included content and examples that might not be 
understood by some parents. 

4. Discussion 

Practitioner social validity judgments of the ECTA Center performance checklists and parent practice guides, and 
suggestions for improving the checklists and practice guides, proved extremely valuable for making improvements in 
the early childhood intervention products and materials. Changes to the checklists and practice guides made in response 
to the findings from the first field test were reflected in the changes in social validity ratings and suggestions from the 
second two field tests as evidenced by the improvement indices for the between field test contrasts (What Works 
Clearinghouse, 2014). Results showed that a rather short field test survey which included both closed-ended and 
open-ended questions yielded information that was helpful in reworking the checklists and practice guides to improve 
their acceptability and likelihood of adoption and use (Foster & Mash, 1999). Results also showed that both prior to and 
after changes were made to the performance checklists and practice guides, the majority of social validity indicators 
were judged as acceptable and important for working with young children and their parents. 

Despite the changes that were made in response to the first field test results, there were repeated as well as additional 
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suggestions for improving the checklists, and to a lesser degree for improving the practice guides on the second and 
third field tests. This feedback will be used to make additional changes to the ECTA Center products to improve their 
acceptability and usability. Additional changes will be made on revised checklists and practice guides as indicated from 
the results from the next round of field tests. 

Although beyond the work scope of the ECTA Center, a logical next step would be to evaluate the relationship between 
practitioner social validity judgments of the checklists and practice guides and the fidelity of use of both products in a 
manner recommended by Strain et al. (2012). As noted by these investigators, social validity judgments are important 
because they will likely be correlated with the fidelity of use of an intervention practice. This type of 
"liking-implementation with fidelity relationship” (Strain et al., 2012, p. 197) was found in a study by Dunst et al. 
(2016), where the effects of parents’ social validity judgments of interest-based everyday activity child language 
learning intervention practices were related to parents’ fidelity of use of the practices and indirectly related to children’s 
rates of language development mediated by the fidelity of use of the intervention practices. Therein lies the importance 
of early childhood intervention practices being judged as socially valid, and the need to evaluate practitioners’ beliefs 
about the importance and acceptability of the practices. Despite the call for systematically including social validity 
measures in research and field-test studies (e.g.. Carter, 2009; Leko, 2014; Turan & Meadan, 2011), this has not become 
routine practice despite its contributions to understanding the adoption and use of different kinds of intervention 
practices. Field-test studies like the one described in this paper serve as a model for informing improvements in early 
childhood intervention practices. 

4.1 Implications for Practice 

Performance checklists and practice guides like the ones described in this paper are simple yet effective tools for 
operationally defining different kinds of early childhood intervention practices which are easily used as part of routine, 
everyday interventions with young children and their families. Twenty-nine performance checklists and 67 (32 parent 
and 35 practitioner) practice guides have been developed for different early childhood intervention practices 
(www.ectacenter.org/decrp). The checklists each include key practice characteristics that serve as mnemonic tools for 
practitioners to understand and remember important steps or elements of the practices. The practice guides each include 
activities that have been found to be related to desired child or family outcomes. 

Both the checklists and practice guides are intentionally formatted in the same way because information and material 
that is organized similarly "causes the learner to focus on the meaning of the material thus increasing the depth of 
processing” (B. L. Schwartz, 2014, p. 107). The checklists and practice guides seem especially needed in a field where 
the knowledge base has expanded so rapidly as is the case for early childhood intervention (see e.g., Reichow et al., 
2016). 
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Appendix A 

Family Capacity-Building Practices Checklist 


This checklist includes practices for engaging parents and other family members in using child-level interventions 
to promote child learning and development in ways that strengthen parenting confidence and competence. 

The capacity-building practices are used by a practitioner to promote a parent’s understanding and use of 
everyday activities and routines as sources of child learning opportunities. 


The checklist can be used by a practitioner to plan intervention sessions with parents and other family members. 
The checklist also can be used to do a self-evaluation to determine if practitioner capacity-building practices 
actively involved parents in providing their children everyday learning opportunities. 


Practitioner: 

_Child: 



Date: 

Please indicate which of the practice characteristics 
you were able to use as part of parent and family 
member involvement in providing child learning 
opportunities: 

Seldom 
or Never 
(0-25%) 

Some of 
the Time 
(25-50%) 

As Often 
As I Can 
(50-75%) 

Most of 
the Time 

(75-100%) Notes 

1. Describe the use and benefits of everyday 
activities as sources of child learning 
opportunities 

□ 

□ 

□ 

□ 

2. Illustrate or demonstrate child engagement in a 
variety of everyday activities 

□ 

□ 

□ 

□ 

3. Describe and illustrate the importance of child 
interests and preferences in promoting child 
learning 

□ 

□ 

□ 

□ 

4. Use an everyday activity checklist to have a 

parent select which activities would be easiest for 
the parent to use 

□ 

□ 

□ 

□ 

5. Together with the parent, engage the child in a 
familiar everyday activity 

□ 

□ 

□ 

□ 

6. Illustrate or demonstrate how adult 

responsiveness to child behavior is used to 
promote child learning in everyday activities 

□ 

□ 

□ 

□ 

7. Provide supportive guidance, feedback, and 
suggestions to the parent throughout the 
capacity-building activities 

□ 

□ 

□ 

□ 

8. Together with the parent, identify five or six 
everyday activities that will be used as child 
learning opportunities 

□ 

□ 

□ 

□ 

9. Engage the parent in conversations about which 
activities will be used for child learning and 
which parent responses will be used to promote 
learning 

□ 

□ 

□ 

□ 


This checklist and other ECTA Center products are available at http://www.ectacenter.org. Copyright © 2017 Early 
Childhood Technical Assistance Center. Reprinted with permission. 
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Appendix B 

Early Childhood Intervention Practice Guide 


c 


Everyday Learning Opportunities. 

Famrty capacity-building practice* are used to *upoort and strengthen parent* and other caregiver* abilities to 
provide their children everyday learning opportunities This is accomplished using a number of different strategies for 
supporting and strengthening parents use of everyday activities to promote child learning and development 


Learning Way: Family-Provided Child Learning Opportunities 



Parents are more likely to use intervention practices with 
their children If they understand the benefits of everyday 
child learning opportunities Begn by explaming both the 
purpose and goal of the learning opportunities and the 
specific benefits to young children 

Ibustrate or demonstrate how to engage a child in 
everyday learning activities Video vignettes of parents 
providing their children everyday learning opportunities 
generally work best. Be sure to point out the key 
characteristics of the practice* 

Engage the parenls .n real-life | authentic I activities to 
provide their children everyday learning opportunities The 
more fameiar and routine the acbvitie* are to parents, the 
more likely they wtli feel confident using the activities as 
sources of child learning opportunities 
Focus on two important child and parent behaviors as 
part of everyday child teaming opportunities The first is 
the importance of child in tmesis and pieferences a* 
factors increasing child engagement in everyday activities 
The second ts the role parents' responsiveness to child 

behavior 





in everyday 


activities plays tn supporting child learning 
Provide parent* supportive guidance and feedback 
during and after parents' use of everyday learning 
opportunities with their child Point out which 
charactenstics of the practice were used by the parents 
and descnfee the child benefits of the practices 
Engage the parents in conversations, discussions, or 
review of the*r confidence using the practices Provide 
specific positive comments, feedback and suggestions 
In response to parents' description* to reinforce their 
sense of confidence. 

Jointly identify four or five everyday activities that the 
parents can use to provide learning opportunities for 
their child. The best activities a/e one* that provide the 
child multiple opportunities to engage in interactions with 
people, toys, and other materials, 
identify things you can do to provide the parents regular, 
ongoing support to encourage the continued use of the 
practices. It is important to plan to take time together to 
review and evaluate the learning opportunities to decide 
which activities should be continued, modified, and 
added 


A Quick Peek Felicity is a 3>year-old with multiple disabilities. 
Her mother Emma, tells her daughter s speech therapist that Felicity 
is beginning to show mcreased interest In looking at other people and 
is making more sounds than usual Mom asks the therapist about the 
best times to work on her daughter a speech The therapist describes 
how there are many opportunities throughout the day that can be used 
to encourage Felicity to use sounds in interactions with mom. dad. and 
other family members The therapist use* video she has stored on a 
tablet compoier to show Felicity's mom and dad how other parents 
have used everyday activities to encourage their young children to make sotexts and talk more ' She pomts out 
the children’s high level of interest in the activities and how the parents imitation of their children s sounds gets the 
children to oonbnue making the sound* At the therapist'* next visit. Emma say* that Feticrty now 'talk* up a storm’ 
while she plays lap games with her daughter The therapist asks Emma to show her what this looks like and suggests 
trying to do the same thing in a few more activities It isn t »ong before Emma identifies more than a dozen activities 
during which she can encourage her daughter to make more sound* 


You'll know it’s working if ... 

• Parents use lots of everyday activities for 
child learning 

• Parents are responsive to their children s 
behavior in the activities 

• The children are interacting with people and 
objects in the activities 


Leam more about helping families make use of everyday learning 
opportunities from online resources such as 

• Tcala and Smites la Facilitate family Enufloamam cot me Ei 

Excellence website 


ECTACenier 

Early Childhood Technical Assistance Center yryrtf.ectacentei.o.'a 
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